feat: Generalize metrics and add nullability delta metric by MoritzPotthoffQC · Pull Request #43 · Quantco/diffly

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-10T13:39:32Z

Motivation

I like the concept of metrics to give the user as much information about their data at first glance. I think it would be nice to extend it even further by generalizing metrics to non-numeric columns and provide metrics that describe the two datasets, not only the change. The specific use-case I have for this is to explain how the share of null values in a column changed.

Changes

Introduce metrics subpackage
Move existing metrics to change submodule
Introduce data submodule for the new metric type (not only numeric)
Keep existing behavior for passing callables (numeric only), but additionally support metrics with custom column selector
Add a nullability change metric with format <old> -> <new> (<delta>).
Update docs and CLI accordingly

codecov · 2026-07-10T13:40:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ea4ee01) to head (901590c).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #43   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        14    +3     
  Lines         1018      1052   +34     
=========================================
+ Hits          1018      1052   +34

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR extends Diffly’s per-column summary metrics to support non-numeric columns via an explicit Metric(fn, selector=...) wrapper, and introduces a built-in metric for nullability changes so users can quickly see how null fractions evolve between left/right frames.

Changes:

Allow summary(metrics=...) / testing helpers to accept either a bare metric callable (numeric-only) or a Metric with a Polars selector (any column types).
Add the null_fraction_change preset and expose it as ΔNull% in DEFAULT_METRICS and the CLI.
Update summary formatting + docs, and add tests + generated summary fixtures for the new metric behavior.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
diffly/metrics.py	Adds `Metric` wrapper type, null-fraction change preset, and updates `DEFAULT_METRICS` to include selector-based metrics.
diffly/comparison.py	Accepts `Metric` objects in `summary(metrics=...)` and resolves bare callables as numeric-only.
diffly/summary.py	Allows metrics to render string results in column summaries.
diffly/testing.py	Broadens `metrics` typing/docs to allow `Metric` values in assertion helpers.
diffly/cli.py	Updates CLI help text and passes selector-based preset metrics through to `summary()`.
docs/api/metrics.rst	Documents selector-based metrics and adds `Metric` + `null_fraction_change` to the API docs.
tests/test_metrics.py	Adds unit tests for `null_fraction_change` output (including non-numeric columns).
tests/cli/test_cli.py	Adds CLI coverage for selecting the `ΔNull%` metric preset.
tests/summary/fixtures/metrics_null_fraction/test_metrics_null_fraction.py	New fixture generator test for summaries including `ΔNull%` and a selector-based custom metric.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for `ΔNull%`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

EgeKaraismailogluQC

Thank you Moritz Potthoff (@MoritzPotthoffQC), this was pleasant to review 😄 I only have some cosmetic suggestions, but also one question regarding the definition of the Null% metric:

The current definition compares the mean Null%s to each other. An alternative definition would be to check this on a per-row basis, i.e. to check how often a row that was Null is now not Null, and vice versa. When implementing the metrics feature, I rather had this second variant in mind since it makes use of the matching between each pair of rows, but the variant you implemented is of course also legitimate. My question: is this alternative definition potentially also interesting for your use case, or are you only interested in the variant which you implemented in this PR?

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

…d-nullability

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

diffly/metrics/change.py:73

Renaming DEFAULT_METRICS to DEFAULT_CHANGE_METRICS in diffly.metrics.change is a backwards-incompatible API change for consumers importing the old name. Consider keeping DEFAULT_METRICS as an alias to avoid breaking downstream code.

EgeKaraismailogluQC

Thanks Moritz Potthoff (@MoritzPotthoffQC)!

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-14T14:28:18Z

Waiting for this discussion to be settled before merging

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

init

e97b755

Moritz Potthoff (MoritzPotthoffQC) self-assigned this Jul 10, 2026

github-actions Bot added the enhancement New feature or request label Jul 10, 2026

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 10, 2026 15:46

better output

1e72823

cleanup

1d7bdeb

refactor

a74429f

This was referenced Jul 10, 2026

feat: Improve existing numeric metrics for new output format #44

Closed

ci: Fix docs build #45

Merged

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:28

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:28 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/metrics.py Outdated

Comment thread docs/api/metrics.rst Outdated

Comment thread diffly/comparison.py

Comment thread tests/test_metrics.py

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 16:36

Potential fix for pull request finding

71338c9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fix

cd33f6a

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 14:40

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 14:40 View session

fix

f66bfa0

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/comparison.py Outdated

Comment thread diffly/metrics.py Outdated

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 10, 2026 17:15

Potential fix for pull request finding

db3b168

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into generalize-metrics-add-nullability

eb1c427

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 10, 2026 15:34

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 10, 2026 15:35 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread diffly/summary.py

Comment thread diffly/metrics/data.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 10, 2026 15:42

Moritz Potthoff (MoritzPotthoffQC) requested review from EgeKaraismailogluQC, Marius Merkle (MariusMerkleQC) and Oliver Borchert (borchero) as code owners July 10, 2026 15:42

EgeKaraismailogluQC reviewed Jul 12, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) and others added 2 commits July 12, 2026 22:39

Update docs/api/metrics.rst

2923f2e

Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

Update tests/summary/fixtures/metrics_null_fraction/test_metrics_null…

e1ea6bd

…_fraction.py Co-authored-by: EgeKaraismailogluQC <128645043+EgeKaraismailogluQC@users.noreply.github.com>

quant-ranger Bot and others added 3 commits July 12, 2026 22:40

chore: Update pixi lockfile (#42)

fc0426a

Co-authored-by: quant-ranger[bot] <132915763+quant-ranger[bot]@users.noreply.github.com> Co-authored-by: Marius Merkle <marius.merkle@quantco.com> Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>

review

f5b9117

fix

e374f0d

Moritz Potthoff (MoritzPotthoffQC) commented Jul 12, 2026

View reviewed changes

Comment thread diffly/metrics/data.py

fix

627721e

Moritz Potthoff (MoritzPotthoffQC) requested a review from EgeKaraismailogluQC July 12, 2026 20:57

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as draft July 13, 2026 15:23

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 13, 2026 18:01

Merge remote-tracking branch 'origin/main' into generalize-metrics-ad…

cf686b4

…d-nullability

fix

529b917

fix

6f5671d

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 13, 2026 17:09

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 13, 2026 17:10 View session

fix docs

bfc8d8c

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py

Comment thread diffly/metrics/__init__.py

Comment thread diffly/testing.py

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 13, 2026 17:17

Moritz Potthoff (MoritzPotthoffQC) commented Jul 13, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py

fix docs

bfc03ad

EgeKaraismailogluQC approved these changes Jul 14, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) enabled auto-merge (squash) July 14, 2026 14:27

Moritz Potthoff (MoritzPotthoffQC) disabled auto-merge July 14, 2026 14:27

review

b309e7a

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 15, 2026 13:33

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 15, 2026 13:34 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread diffly/cli.py

fix CLI

901590c

Oliver Borchert (borchero) approved these changes Jul 16, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) merged commit 01cef28 into main Jul 16, 2026
18 checks passed

Moritz Potthoff (MoritzPotthoffQC) deleted the generalize-metrics-add-nullability branch July 16, 2026 10:57

Moritz Potthoff (MoritzPotthoffQC) mentioned this pull request Jul 16, 2026

feat: Add data metrics for numeric columns #46

Open

Uh oh!

Conversation

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

codecov Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Moritz Potthoff (MoritzPotthoffQC) commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Moritz Potthoff (MoritzPotthoffQC) commented Jul 10, 2026 •

edited

Loading

codecov Bot commented Jul 10, 2026 •

edited

Loading